{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "genomelake.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ID1m1CKiWa7z"
      },
      "source": [
        "# Integrating Hub to an Existing Package to boost performance #2\r\n",
        "\r\n",
        "This is an experiment to compare Hub's performance in real life packages. We select a package that currently uses a TileDB backend and record its performance, then the backend is ported to Hub, and performance of the new backend is recorded on the same tests as before. Ideally, the Hub backend should outperform the previous TileDB backend.\r\n",
        "\r\n",
        "### Name of package: genomelake\r\n",
        "\r\n",
        "Link to original package: https://github.com/kundajelab/genomelake\r\n",
        "\r\n",
        "Link to modified package: https://github.com/DebadityaPal/genomelake\r\n",
        "\r\n",
        "### Why this package?\r\n",
        "\r\n",
        "The motivation behind selecting \"genomelake\" as the package is that we wanted to compare Hub's performance to that of TileDB, before branching out to other similar alternatives. For ease of portability the package selected had to be smaller in size so that a single programmer can quickly understand the working mechanism of the package and start to port the backend to Hub, secondly the package had to use a TileDB backend since that is the package we want to compare Hub to. The intersection of these requirements produced a couple of OSS packages, but few of them were outdated, few more had errors which could not be resolved and so on. \"genomelake\" was the optimal choice as it was relatively small in size, ran without any inherent errors and was also relatively better known than the other packages in terms of GitHub Repo stars.\r\n",
        "\r\n",
        "## The Experiment Follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mn8SN6lZkXg0"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount(\"/content/drive\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFzKyoU5kmnt"
      },
      "source": [
        "## Fetching the Data\n",
        "\n",
        "We fetch a NCBI protein sequence dataset of a particular species of yeast and use that as our testing dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-LRJpPhtun-x"
      },
      "source": [
        "!wget \"https://drive.google.com/uc?export=download&id=1rF5kLOEBdAi7qo3EYZoI0Ixy7snBamsf\" -P \"/content/drive/MyDrive/Genomelake/Data\"\n",
        "!mv \"/content/drive/MyDrive/Genomelake/Data/uc?export=download&id=1rF5kLOEBdAi7qo3EYZoI0Ixy7snBamsf\" \"/content/drive/MyDrive/Genomelake/Data/yeast_2.fa\"\n",
        "!mkdir \"/content/drive/MyDrive/Genomelake/Repo\"\n",
        "%cd \"/content/drive/MyDrive/Genomelake/Repo\"\n",
        "!git clone https://github.com/DebadityaPal/genomelake"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBZwDntPkZed"
      },
      "source": [
        "##Installation\n",
        "\n",
        "The following cell will install all the dependencies and the package itself, users will have to restart the colab environment after running this cell if they are running it on Google Colab"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ckbT5PTYtLhy"
      },
      "source": [
        "%cd \"/content/drive/MyDrive/Genomelake/Repo/genomelake\"\n",
        "\n",
        "!pip install  hub\n",
        "!pip install pyBigWig\n",
        "!pip install bcolz\n",
        "!pip install pybedtools\n",
        "!python setup.py install"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "plWwCv9Lk1j3"
      },
      "source": [
        "(Only for Google Colab environment)\n",
        "\n",
        "If the runtime enviroment was restarted, users can run the notebook from the next cell onwards, previous cells dont need to be executed provided they were executed before the restart."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ezkZvMinzE-A"
      },
      "source": [
        "%cd \"/content/drive/MyDrive/Genomelake\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnvpLVuBk6nW"
      },
      "source": [
        "## Testing\n",
        "\n",
        "We will be testing a particular function from the repository named \"extract_fasta_to_file\", this function takes a FASTA file and converts it into a dataset with the backend of choice given by the user. This function can effectively check the writing efficiency of the backends."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOgi2WIn0S-6"
      },
      "source": [
        "from genomelake.backend import extract_fasta_to_file\n",
        "import time\n",
        "\n",
        "genome_fasta = \"./Data/yeast_2.fa\"\n",
        "genome_data_directory = \"./Data/yeast_2\"\n",
        "\n",
        "def time_hub():\n",
        "  start = time.time()\n",
        "  extract_fasta_to_file(genome_fasta, genome_data_directory+\"_hub\", mode=\"hub\")\n",
        "  end = time.time()\n",
        "  print(\"Time taken by Hub: \", end-start)\n",
        "\n",
        "def time_tiledb():\n",
        "  start = time.time()\n",
        "  extract_fasta_to_file(genome_fasta, genome_data_directory+\"_tiledb\", mode=\"tiledb\")\n",
        "  end = time.time()\n",
        "  print(\"Time taken by tiledb: \", end-start)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xzFYKTuj6nBG"
      },
      "source": [
        "time_tiledb()\r\n",
        "time_hub()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RmkEnrhZlZjg"
      },
      "source": [
        "##Clearing out the Space\n",
        "\n",
        "**IMPORTANT:**\n",
        "\n",
        "Run this cell very carefully and only after checking the path. This cell will delete the root folder of the package that was created when the initial cells were run. Changing the path can have unwanted deletion of other files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hPy-c7by7OhG"
      },
      "source": [
        "!rm -r \"/content/drive/MyDrive/Genomelake\""
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zk1-HnCFgQUF"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}